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1 + This application is submitted in the name of the following inventor(s): 

2 

3 Inventor Citizenship Residence City and State 

4 CHERITON, David Canada Palo Alto, California 

5 

6 The assignee is Cisco Systems, Inc. , having an office at 170 West Tasman 

7 Drive, San Jose C A 95 1 34. 

Title of the Invention 
M-trie gh^- Ex lci uk d T RIE Based Packet -Lookup Processing 
Background of the Invention 

Field of the Invention 

This invention relates to use of an expanded data structure (referred to 
18 herein as M-trie Plus) for packet processing. 
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1 2. Related Art 
2 

3 In a computer network, a router or switch operates to receive messages at its 

4 input interfaces and to send messages from its output interfaces. In performing these 

5 tasks, the router or switch must generally determine which output interface (if any) is ap- 

6 propriate for forwarding the message. When making this determination, the router or 

7 switch is responsive to the destination IP address (for multicast packets, the router or 

8 switch is also responsive to the source IP address). Similarly, the router or switch can 

9 also be responsive to the source IP address for policy routing. 

10 

1 1 A first method of identifying an outbound interface for a particular packet is 

12 to look up the IP address on the packet in a TRIE data structure, as described in the In- 

1 3 corporated Disclosures. 

14 

15 In a known system described in the Incorporated Disclosures, the router or 

16 switch uses each of the four bytes of the destination IP address to perform a lookup in a 

17 branching table having a set of up to 256 possible branching entries in a TRIE data 

18 structure, terminating in a leaf node in the TRIE upon or before having performed the 

19 lookup for each byte of the destination IP address. Thus, it is possible to determine an 

20 appropriate output interface for the destination IP address by going through no more than 

21 four cycles on the TRIE before coming to a leaf node that specifies the actual handling 

22 instructions for the packet. 
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1 One drawback to TRIE data structure processing is that it is limited to proc- 

2 essing the destination IP address. It would be desirable to expand on this so as to single 

3 out other aspects of the packet header beyond the four bytes specifying the destination IP 

4 address. 

5 

6 A second drawback to the use of a TRIE for lookups is that access control 

7 list (ACL) processing remains separate from routing. In some embodiments, ACL proc- 

8 essing is driven by software. Since this process is relatively slow when compared to high- 

9 speed routers, ACL processing slows the overall rate at which data packets are forwarded. 
10 Depending upon the length of the ACL criteria, router speeds can drop 70% or more. 

11 

12 A third drawback to use of a TRIE is that the nodes and leaves of the TRIE 

13 generally do not provide adequate information to direct multicast routing. 

14 

15 Accordingly, it would be desirable to provide an improved technique for 

16 looking up information contained in a packet header relevant to routing and access con- 

17 trol. This is achieved in an embodiment of the invention which is a novel and nonobvious 

18 expansion on an expanded TRIE structure, herein called an M-trie Plus data structure. In 

19 addition to providing unicast routing and access control list processing, an M-trie Plus 

20 data structure can be used with techniques for multicast routing, ACL processing, CoS 

21 (class of service) processing, QoS (quality of service) processing, and the like. 

22 
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1 Summary of the Invention 
2 

3 In a first aspect of the invention, different aspects of the packet header are 

4 singled out for attention, rather that just the four byte IP destination address. This allows 

5 the M-trie Plus to perform functions that trie data structures were unable to do. Current 

6 TRIE structures distinguish only between the leaf and node type elements and are used 

7 only for routing. The M-trie Plus extends this and includes different information in the 

8 nodes of the TRIE which enables matching and branching on different header fields. The 

9 basic building block of all M-trie Plus nodes is an oppointer. The oppointer includes an 

10 address and an opcode. In a preferred embodiment, the address included in an oppointer 

11 is the address for the next node. The opcode included in an oppointer describes what ac- 

12 tion the router or switch has to do on the packet label to select the next oppointer leaf on 

13 the M-trie Plus. If an oppointer points to the 8 bit termination leaf, the lookup is termi- 

14 nated. High speed packet header processing is achieved by the multiple pipelined threads 

15 of the M-trie Plus engine (MPE) and a wide memory bus. 

16 

17 In a second aspect of the invention, the ACL of a configuration file in a 

18 router or switch is compiled into an ACL-M-trie Plus data structure which is located in 

19 the memory of the router or switch. This has the effect of merging routing and ACL 

20 processing in a single device. The M-trie Plus data structure is traversed with respect to 

21 information included in the packet header, thereby determining whether a packet should 

22 be dropped or forwarded. ACL lists are defined in the configuration file of the router or 
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1 switch. In a preferred embodiment, there are two forms of access list in the IOS: the 

2 standard ACL and the extended ACL. Standard lists are used to control traffic based on 

3 one or more source IP addresses. The extended access list provides a finer granularity in 

4 controlling traffic. ACL definitions provide a set of criteria that are applied to each 

5 packet that is processed by the router or switch. The router or switch decides whether to 

6 forward or drop each packet based on whether or not the packet matches the access list 

7 criteria. Typical criteria defined in ACLs are source addresses, destination addresses or 

8 upper-layer protocols of the packet. 

O 9 

yho In a third aspect of the invention, the M-trie Plus structure can map a multi- 

~hi cast packet header by a sequence of nodes that match a destination address or source ad- 

J 12 dress. Each physical port uses the M-trie Plus with the first level nodes matching on the 

Oi3 first 8 bits of the destination address, the second level nodes matching on the second 8 

J^14 bits of the destination address and so on, at each level the nodes correspond to multicast 

o 

q 15 addresses. In other embodiments, the nodes can compare more than just 8 bits. 

16 

17 In a preferrecTfeqibodiment, the opcode included in a node can specify other 

18 operations, such an instruction to cbmmre bytes in the packet header with bytes in a 
V) CAM (contest addressable memory) or to direct^certain types of packets (for example, 
2p voice traffic) to a specified output interface. 
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Incorporated Disclosures 

The inventions described herein can be used in conjunction with inventions 
described in the following applications: 

• Application Serial Number 08/886,900, in the names of Darren Kerr and Barry Bruins, 
titled "Network Flow Switching and Flow Data Export", assigned to the same as- 
signee, attorney docket number CIS-021, and all pending cases claiming the priority 
thereof. 

• Application Serial Number 08/655,429, filed May 28, 1996, in the names of Darren 
Kerr and Barry Bruins, titled "Network Flow Switching and Flow Data Export", as- 
signed to the same assignee, attorney docket number CIS-016 and all pending cases 
claiming the priority thereof 

• Application Serial Number 08/581,134, filed December 29, 1995, in the names of 
David Cheriton and Andy Bechtolsheim, titled "A Method for Traffic Management, 
Traffic Prioritization, Access Control and Packet Forwarding in a Datagram Computer 
Network", assigned to the same assignee, attorney docket number CIS-019. 
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1 Lexicography 
2 

3 • Access control lists (ACL) - as used herein, the term "access control lists" is syn- 

4 onymous with the term "traffic filter". In general, it includes a list of the services 

5 available on a server, each with a list of the hosts permitted to use the service. ACLs 

6 can define the accessibility of networks and hosts through a router or switch because 

7 they determine whether packets are dropped or forwarded at the router or switch inter- 

8 faces. 

n 9 

m 10 • Content addressable memory (CAM) - as used herein, the terms "CAM" and 

V1 1 1 "Content Addressable Memory" include devices used in a computer system for storing 

(i 12 and retrieving information. CAMs have the advantage that they can rapidly flag cer- 

Q 13 tain data by linking associated data values with known tags; thus making it possible to 

0 14 perform rapid lookup of the associated data values once the tag is known. 

n 15 

16 • CoS - as used herein, the term "CoS" refers to the class of service such as voice traf- 

17 fic, email traffic, wireless traffic. Many network protocols allow packet headers to in- 

18 elude CoS information. 

19 

20 • QoS - as used herein, the term "QoS" refers to the quality of service and includes the 

21 performance properties of a network service such as throughput, transit delay and pri- 
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ority. Many network protocols allow packets or data streams to include Qos require- 
ments. 

• TRIE- as used herein, the term "TRIE " refers to a tree-like data structure that is used 
to determine the routing of data packets by looking to information in the packet header 
and matching it to information included in the node of the data tree. 

• M-trie Plus - as used herein, the term "M-trie Plus" includes an extension of the ex- 
isting TRIE. Instead of mere matching, every node in the tree includes an address and 
an opcode. This additional information allows the router or switch to look up packet 
headers and perform simple instructions related thereto relatively rapidly. 

• M-trie Plus Engine - as used herein, the term "M-trie Plus Engine" is a multi- 
threaded processor included in a router or switch that services a queue of packet head- 
ers. 

• Oppointer - as used herein, the term "oppointer" derives from the words "opcode" 
and "pointer". An oppointer is a micro-code structure to the router or switch that in- 
cludes a 10 bit opcode and a 22 bit address. Each node in the M-trie Plus data struc- 
ture is specified by an oppointer. 

• ACL list- as used herein the term "ACL list" refers to a set of criteria which are ap- 
plied to each packet that is processed by the router or switch. The router or switch de- 
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cides whether to forward or drop each packet based on whether or not the packet 
matches the access list criteria. 

• Standard ACL list - as used herein, the term "standard ACL list" includes lists of 
qualifiers that are used to control traffic based on one or more source IP addresses. In 
a preferred embodiment, the IP address qualifier is a 32 bit quantity in dotted decimal 
format. 

• Extended ACL list - as used herein, the term "extended ACL list" provides a finer 
granularity than the standard ACL lists, with respect to criteria used in controlling 
traffic. 

• TRIE - as used herein, the term "TRIE" includes data structures that store elements in 
a tree, including roots, leaves and branches. The path from the root to the leaf is de- 
scribed by a key. 

• Flow label - as used here, the term "flow label" describes the collection of fields used 
to identify and classify fields in the packet header, including, without limitation IP 
source address, destination address, protocol type and layer 4 port numbers. 
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Brief Description of the Drawings 



Figure 1 shows a block diagram of a system that includes an M-trie Plus 
data structure and a set of oppointers used in routing data packets. 

Figure 2 shows a data structure showing M-trie Plus subtrees and an op- 
pointer. 

Figure 3 shows a process flow diagram of a method for using a system that 
includes an M-trie Plus data structure and a set of oppointers used in routing data packets 
and access control. 

Detailed Description of the Preferred Embodiment 

In the following description, a preferred embodiment of the invention is de- 
scribed with regard to preferred process steps and data structures. Those skilled in the art 
would recognize after perusal of this application that embodiments of the invention can 
be implemented using one or more general purpose processors or special purpose proces- 
sors or other circuits adapted to particular process steps and data structures described 
herein, and that implementation of the process steps and data structures described herein 
would not require undue experimentation or further invention. 
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System Elements 

Figure 1 shows a block diagram of a system that includes an M-trie Plus 
data structure and a set of oppointers used in routing data packets. 

A system 100 includes at least one source device 1 10, a data stream 120, at 
least one routing or switching device 130 and at least one destination device 140. 

The source device 110 includes any device on a network of networks (an 
internet) that is identified by its IP (internet protocol) address. 

The data stream 120 includes one or more data packets 121 that travel from 
a source device 110 to one or more destination devices 140. Each data packet 121 in- 
cludes a packet header 122, which includes information for routing the data packet 121. 
Although the preferred embodiment of data stream 120 is a unidirectional stream, other 
embodiments may be bi-directional. 

The destination device 140 is any device on a network that can be specified 
by its IP address. 
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THE ROUTER OR SWITCHING DEVICE 

3 The routing or switching device 130 processes data packets 121 from at 

V4 least one source device 1 rQ and directs them to at least one destination device 140. The 

5 routing or switching device 136^ includes one or more input interfaces 131, a routing 

K processor 132, an M-trie plus engineN^, an M-trie data 200 and a set of output inter- 

7 faces 134. 



n 9 Packets are received one or more input interfaces 131 and processed by the 

fJho routing processor 132. The routing processor 132 includes a processor and memory for 

If! 11 performing the process steps described herein, and may include specific hardware con- 

§jji2 structed or programmed for performing the process steps described herein. In a preferred 

O13 embodiment, the routing processor 132 includes a high-performance, highly integrated 

3 14 router chip set using shared memory implemented by multi-bank pipelined SDRAM, 

g 15 Such embodiments are capable of supporting a plurality of OC-48 ports, a plurality of Gi- 

16 gabit ethernet ports and a plurality of 10/100 Megabit ethernet ports. 

17 

18 The M-trie plus engine 133 is a multi-threaded processor and memory that 

19 services a queue of packet headers 122 and determines which one or more output inter- 

20 faces 134 from the set of output interfaces 134 that a particular packet is destined for. 

21 The memory of the M-trie plus engine 133 includes an M-trie plus data structure 200. 
22 
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Figure 2 shows a block diagram showing M-trie Plus subtrees and oppoin- 



2 



ters. 



3 





6 



plurality 



The M-trie plus data structure 200 includes a tree having a root node 205, a 
of inferior nodes 210 and a tbmiinal leaf node 215. 



7 



The root node 205, inferior nodes 210 and terminal leaf node 215 include 



8 an oppointer 220. Each oppointer 220 includes an address 225 and an opcode 230. The 

m 9 address 225 specifies a table and a location in a table where further instructions regarding 

J! 10 the packet 121 are found. The opcode 230 includes instructions concerning what to do 

y]n with the packet 121, including what operations the M-trie Plus engine 133 must execute 

■2 12 on the packet header 122 to cause it to compute and fetch the next oppointer. For exam- 

.«|13 pie, an opcode 230 could include instructions to lookup the destination IP address or the 

Q14 source IP address. Much information can be rapidly processed as the lookup process 

gi5 traverses the plurality of inferior nodes 210 until a terminal leaf node 215 is reached and a 

16 decision to drop or pass the packet 121 is made. 



19 over M-trie data structures as described in the Incorporated Disclosures. In addition to 

20 specifying an output interface 134, this information can be used to direct multicasting, 

21 access control, CAM lookups and numerous other processes that would be obvious to one 

22 skilled in the art. Moreover, unlike existing M-tries, the M-trie plus technique can be 



17 



18 



Information included in the oppointer 220 provides substantial advantages 
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used with any four bytes of the packet header, not just the four bytes that specify the des- 
tination source. 

Depending on the type of information in the opcode 230, the root node 205, 
inferior nodes 210 and terminal leaf node 215 can be categorized as demultiplexing (de- 
nux) nodes, matching modes, hashing nodes or specialized nodes. 

Demultiplexing nodes demultiplex into different M-trie plus branches based 
on the value of the selected byte in the packet header. 

Matching nodes compare the given byte value of the packet label to given 
node data. A match node matches on one value and provides two subnodes correspond- 
ing to "match" and "not match". The result indexes the next oppointer 220. These 
matching nodes can also compare more than a byte. 

Hashing nodes hash into different M-trie plus branches based on the value 
of the selected byte in the packet header 122. 

Specialized nodes perform .operations that cannot be performed by other 
nodes. These specialized operations include termination of the lookup process. Unlike 
M-trie termination, which relies upon a LSB-bit special mechanism to distinguish be- 
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tween nodes and leaves, termination of a lookup in an M-trie plus data structure 200 relies 
upon a termination leaf containing an 8 bit term instruction. 

In a preferred embodiment, one of the subtrees of the M-trie Plus data 
structure 200 includes an M-trie Plus - ACL data structure. Compiling this data into a 
subtree rather than a standard M-trie minimizes the lookup count and memory usage. 



Method of Use 



Figure 3 shows a process flow diagram of a method for using a system that 
includes an M-trie Plus data structure and a set of oppointers used in routing data packets 
for access control. 

The metho&300 is performed by the systems 100 and 200. Although the 
method 300 is described serially the steps of the method 300 can be performed by sepa- 
rate elements in conjunction or parallel, whether asynchronously, in a pipelined manner, 
or otherwise. In broad overview, the method 300 can include routing of packets, multi- 

\ 

casting, deciding whether packets can be dropp^das a function of QoS or CoS and other 
aspects related to processing of packet headers. 

At a flow point 300, the systems 100 and 200 are ready to begin processing 
and routing data packets 121 . 
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1 At a step 310, at least one source device 110 transmits one or more data 

2 packets 121. The data packets 121 are input to the routing or switching device 130 at one 

3 or more input interfaces 131. 

4 

5 At a step 315, the M-trie Plus engine 133 accesses the root node of the M- 

6 trie Plus data structure 150 and initializes its forwarding state. 

7 

8 At a step 320, the M-trie Plus engine 133 determines whether the process- 



9 ing is complete, as indicated by reaching a terminal node. If processing is complete, the 

10 method 300 proceeds at step 340. If processing is not complete, the method 300 proceeds 

11 at step 330. 

12 

13 At a step 330, the M-trie Plus engine 133 extracts the data field from the 

14 packet 121 specified by the current node opcode. The opcodes in the oppointer (that is, 

15 the 10 bit opcode 230) can refer to any portion of the packet flow label, or more gener- 

16 ally, to any field in the packet. For example, the opcode may refer to the IP address for 

17 the source device 110, the IP address for the destination device 140 or the protocol type 

18 for the packet 121. 

19 v. 

20 For example (withoutTimitation) a first oppointer can have an opcode 110 

21 specifying match on protocol field and a p^k^ter (that is address 225) to another node in 

22 the M-Trie Plus data structure 150. This node mziy^have an opcode 110 that specifies 
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1 hash and demux on the\ast byte of the source address. The next oppointer can specify to 

2 multiplex on the second byte of the destination address. 



4 In this way, the router can traverse an access control list, if such a list is im- 

5 posed) (i.e. whether the intended destination device is authorized to receive a particular 

6 packet), QoS parameters (whether a percentage of packets in a data stream for one of one 

7 or more destination devices should be dropped) and other parameters that would be obvi- 

8 ous to one skilled in the art of packet processing. In a preferred embodiment, the lookup 
i 9 could be either relatively simple and involve as few as a single byte or be relatively corn- 
el o plex and involve several hundred oppointers. 



M2 




In a step 335, W M-trie Plus engine 133 accesses the node of the M-trie 



pi3 Plus data structure 150 that is determined by the address in step 330. The method 300 
□ 14 proceeds at step 320. 

In a step 340, the data packet 12l\is passed to one or more output interfaces 
135 or dropped. The decision to pass or drop the packet 121 is responsive to information 
1 8 contained in the terminal leaf node 215. 

19 
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Alternative Embodiments 



Although preferred embodiments are disclosed herein, many variations are 
possible which remain within the concept, scope, and spirit of the invention, and these 
variations would become clear to those skilled in the art after perusal of this application. 
In particular, the invention can be applied to matching and classification of HTTP head- 
ers. 



Express Mailing No. EL524780790US 18 



